How to measure gender bias in machine translation: Real-world oriented machine translators, multiple reference points

In this paper—as a case study—we present systematic study of gender bias in machine translation with Google Translate. We translated sentences containing names occupations from Hungarian, language gender-neutral pronouns, into English. Our aim was to fair measure for by comparing the translations real-world oriented non-biased translator. When assessing bias, we used following reference points: (1) distribution men and women among both source target countries, as well (2) results Hungarian survey that examined if certain jobs are generally perceived feminine or masculine. also studied how expanding adjectives referring affects pronouns. As result, found against genders, but biased much more frequent. Translations closer our perception than objective occupational statistics. Finally, have greater effect on adjectives.